The n th-Order Bias Optimality for Multichain Markov Decision Processes

The paper proposes a new approach to the theory of Markov decision processes (MDPs) with average performance criteria and finite state and action spaces. Using the average performance and bias difference formulas derived in this paper, we develop an optimization theory for average performance (or gain) optimality, bias optimality, and all the high-order bias optimality, in a unified way. The approach is simple, direct, natural and intuitive; it does not depend on Laurent series expansion and discounted MDPs. We also propose one-phase policy iteration algorithms for bias and high-order bias optimal policies, which are more efficient than the two-phase algorithms in the literature. Furthermore, we derive the high-order bias optimality equations. This research is a part of our effort in developing sensitivity-based learning and optimization theory. The new insights provided by this approach may lead to some new research directions such as on-line learning, performance derivative based optimization, and potential or high-order potential aggregations.